Add request phase timing: Server-Timing subtimings and access telemetry - #1074
Add request phase timing: Server-Timing subtimings and access telemetry#1074jevansnyc wants to merge 26 commits into
Conversation
* Add request phase timing design spec (Server-Timing subtimings + access telemetry) * Address review round 1: freeze point, template-cache naming, snapshot semantics, KV scope, geo carry, route template, sink confirmation, sampling and query model, config rollback * Address review round 2: auction-wait placement modes, conservative private-only header emission, non-null sorting key with service identity, coarse publisher route template, telemetry snapshot and outage behavior, tinybird flag decoupling, adapter phase semantics * Add request phase timing implementation plan * Address engineer review: KV timing decorator, try_lock sampling, route metadata extension, adapter-derived env, typed template-cache state, adapter-owned emission context, per-mode delivery semantics, Axum outer wrapper
…ite-back in middleware Three final-review fixes for access telemetry correctness: - Normalize the HTTP method to an allowlist (GET/HEAD/POST/PUT/DELETE/ PATCH/OPTIONS, else "other") inside access_event_row, so a client- controlled extension-method token can never inflate the LowCardinality method column, regardless of which adapter builds the row. - Guard emit_access_telemetry_after_send against snapshots carrying a degraded sample_rate of 0.0 (captured on the app-state-build-failure fallback path), which could otherwise be sampled in by freshly reloaded settings and corrupt the sum(1.0/sample_rate) volume estimator. - Mirror the geo lookup write-back from apply_entry_point_finalize_headers into FinalizeResponseMiddleware::handle, so a middleware-finalized response that resolved geo via fallback carries the resolved GeoLookupState for the access-telemetry snapshot instead of showing country "unknown".
|
Post-review addition from the first live full-stack test (stackpop.com staging property): commit 7cf7d86 adds JSONPaths to every access_logs_raw column and replaces the event_date DEFAULT column with a toDate(event_ts) sorting-key expression. The Events API rejects NDJSON ingestion into a datasource without JSONPaths (400, discovered live; the confirmed-delivery check surfaced it via the drop warning), and once any column has a path every column needs one, which a DEFAULT column the producer never sends cannot satisfy. Spec section 9 updated in the same commit. Verified end to end: rows now flowing guest -> Events API -> ClickHouse with correct phase attribution and route-template normalization. |
prk-Jr
left a comment
prk-Jr
left a comment
There was a problem hiding this comment.
Review summary
Verdict: Request changes.
The timing core is careful, well-reasoned work, and the Tinybird schema discipline is actually better than the PR body claims (verified below). The blocker is that this ships a reproducible crash-on-every-request regression on the Cloudflare adapter — with a one-line fix — plus a second instance of the same bug that CI structurally cannot catch, and a route normalizer that lets UUIDs, reset tokens and article slugs into a 30-day dataset.
Verification performed
Reviewed at head 7cf7d867 in a scratch worktree. Local gates, all run there:
| Gate | Result |
|---|---|
cargo fmt --all -- --check |
PASS |
cargo clippy-fastly / clippy-axum / clippy-cloudflare-wasm |
PASS |
cargo test-fastly |
PASS (2235 + 186 + 21 + 4 + 2) |
cargo test-axum |
PASS (25 + 14 + 1) |
npx vitest run |
PASS — 45 files, 871 tests, 0 failures |
gh pr checks 1074: 19 checks, 1 failing — integration tests. It is not in the branch-protection required set, so it will not block the merge button, but it is a genuine regression (root-caused inline).
Blocking
- CRITICAL —
std::time::Instantpanics onwasm32-unknown-unknown; every publisher request on Cloudflare traps. Confirmed by reproducing the failing integration test locally, capturing the workerd stack trace, applying the fix, and re-running green (15.34s fail → 1.16s pass). - HIGH — Two more
Instant::now()calls on the auction path inpublisher.rs. The integration fixture has[auction] enabled = false, so CI stays green even after finding 1 is fixed, while Cloudflare does dispatch auctions in production. - HIGH —
publisher_route_templateadmits UUIDs, opaque tokens and full article slugs into a 30-day dataset, contradicting the module's own doc comment.
Verified praise
- Schema alignment is exactly right. Parsed all four artifacts: 26 datasource columns, 26 producer keys, 26 fixture keys, 26
FORWARD_QUERYcolumns — identical names and identical order, zero set difference. try_lock-only is real. 7 lock acquisitions inrequest_timing.rs, alltry_lock, zero.lock(). No method calls another while holding its guard, so no re-entrant self-deadlock.- Saturating arithmetic is complete —
saturating_addinrecord/record_auction_wait/CountingWriter::write,try_from(..).unwrap_or(MAX)in bothduration_msandrecord_buffered_delivery. - The Server-Timing cache-control gate holds on every emit path. Both emitters funnel through
append_server_timing_if_private, andcache_control_value_has_directiveis exact-name and quote-aware (not-private/no-storeycorrectly do not match). On Fastly the call sits afterapply_terminal_response_effectsand both finalize passes, soCache-Controlis settled and nothing mutates it afterwards. [observability]back-compat is sound.Settingscarriesdeny_unknown_fieldsand the pushed blob is a serde serialization ofSettings, soskip_serializing_if = "ObservabilitySettings::is_default"genuinely keeps a default table out of the blob.TinybirdSettingshas nodeny_unknown_fieldson either side, so the newauction_enabledkey does not break rollback either.- The hand-rolled Axum
serveis behaviourally equivalent to the upstream helper it replaces (Stores::default()makes every store-attach branch a no-op; the rest is mirrored exactly).
Audit of the PR body's stated limitations
Most disclosures check out. Four do not:
| Disclosure | Verdict |
|---|---|
| "The 27 vitest failures ... will block the JS CI gate" | Stale. vitest is green on CI; locally 871 tests, 0 failures. |
| "Cloudflare and Spin collect but do not emit in v1" | Badly understated. Spin is fine (wasm32-wasip1, std Instant works). Cloudflare does not collect — it traps on every publisher request. |
"sorting key (event_date, service_id, ...)" |
Inaccurate. Actual key is toDate(event_ts), service_id, ...; event_date is not a column at all. |
| Failing Cloudflare integration job | Unmentioned, and failing on all three runs of the branch. |
Accurate as written: deploy/rollback ordering and the older-binary rejection mechanism (verified end to end), column-for-column schema alignment, the inspection-only Tinybird caveat and incompatible sorting-key change, Axum header-only semantics, DeliveryResult collected-but-unemitted, and the buffered-path request_elapsed_ms placement.
One nuance: "infallible by construction ... no panics" is accurate on locking and arithmetic, but slightly overstated given the unchecked phases[index] array access (noted inline, non-blocking).
std::time::Instant::now() panics on wasm32-unknown-unknown, so every publisher request on the Cloudflare adapter trapped when the timing collector was constructed, and the two auction-wait sites would trap once an auction dispatched. web_time re-exports std's Instant on every other target, so Fastly, Axum, and Spin behavior is unchanged. The publisher.rs sites are qualified locally because that module's std Instant import still serves the pre-existing template-cache sites, which are out of scope here.
The character allowlist alone does not bound identity: [a-z0-9_-] is exactly the alphabet UUIDs, hex ids, reset tokens, and article slugs are built from, and truncating to 32 characters still leaves a globally unique prefix. A first segment now rejects whole to /other/* when it exceeds 32 characters or carries more than 7 ASCII digits, alongside the existing charset rejection. Year archives and hyphenated section names still pass. Extends the adversarial tests to the publisher-fallback path with UUID, hex-id, token, and slug shapes, and fixes the stale event_date reference in the row-builder doc.
- Gate building the access snapshot on tinybird.enabled and access_enabled, threaded through SendContext: a disabled deployment (the default) no longer pays env reads and String allocations on the pre-send path. DeliveryOutcome.snapshot becomes Option and the emitter treats None as nothing to send. - Classify asset-fallback responses as route_class asset with the operator-configured route prefix as the template, instead of landing in the other/unknown bucket alongside 404s. - Pin Phase::index() to PHASE_COUNT with a uniqueness-and-bounds test so a future variant fails the suite instead of panicking at runtime. - Drop the tautological sampled-out emission test; the 0.0-rate behavior is covered by sampled_in_boundary_rates_are_unconditional. - Clarify that the local dev config env var name genuinely triples trusted_server_config (prefix, store, key) rather than reading as a find/replace mistake.
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
👍 Reviewed the exact head revision 38043d7464362d44519153a09fe850bacc256b58. The overall implementation direction is solid and all reported CI checks pass. I left seven actionable inline findings: three P2 telemetry-correctness issues and four P3 design/test-cleanup issues. There are no P0 or P1 findings.
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
🔧 Superseding my prior approval for 38043d7464362d44519153a09fe850bacc256b58: please address the seven inline findings already posted in the preceding review, particularly the three P2 telemetry-correctness issues, before merge. The existing inline threads remain the actionable review details and are not duplicated here.
prk-Jr
left a comment
prk-Jr
left a comment
There was a problem hiding this comment.
Summary
Substantial, carefully-built observability layer: 27 files, ~6.5k insertions, with a
freeze-point design that reads typed response extensions rather than the headers they
back, symmetric geo write-back on both finalize sites, and adversarial route-template
tests. All 19 GitHub checks pass and I reproduced every CI gate locally (2450 Fastly +
40 Axum + 39 Cloudflare + 79 Spin + 13 parity Rust tests, 871 JS tests).
One blocking finding, and it is a spec-conformance question rather than a crash risk:
publisher_route_template does not meet section 9 of
docs/superpowers/specs/2026-08-24-request-phase-timing-design.md for single-segment
paths. Everything else below is non-blocking.
I also checked the access sampler for modulo bias and it came back clean — see the
note in the cross-cutting section, since it is the kind of thing worth recording as
verified rather than leaving as an open question.
4 of the inline comments below carry a one-click GitHub
suggestion. I applied all
four together in a scratch worktree at this head and verified them:
cargo fmt --all -- --checkPASS,cargo clippy-fastlyPASS (0 warnings),
trusted-server-corelib 2238 passed / 0 failed,trusted-server-adapter-fastly
185 passed / 0 failed.
Blocking
🔧 wrench
publisher_route_templatekeeps single-segment article slugs verbatim, against spec section 9 — inline atcrates/trusted-server-core/src/access_telemetry.rs:189
Non-blocking
🤔 thinking
Server-Timingis an unrestricted client-visible latency oracle when enabled — inline atcrates/trusted-server-core/src/request_timing.rs:285
♻️ refactor
HEADER_PHASESis a second hand-synced list with no test — inline atcrates/trusted-server-core/src/request_timing.rs:419#[allow(dead_code)]on the wholeDeliveryOutcomemasks two live fields — inline atcrates/trusted-server-adapter-fastly/src/main.rs:571
🏕 camp site
route_class_renders_snake_caseomits the newAssetvariant — inline atcrates/trusted-server-core/src/access_telemetry.rs:526
⛏ nitpick
- Test name asserts the opposite of what it tests — inline at
crates/trusted-server-core/src/access_telemetry.rs:439
👍 praise
- Freeze point reads extensions, not headers — inline at
crates/trusted-server-adapter-fastly/src/main.rs:797
Cross-cutting
-
✅ Checked and clean: the access sampler is not modulo-biased.
entropy = since_epoch.as_nanos() as u64 ^ outcome.bytesthenentropy % 1_000_000 < rate * 1e6looks like it could starve whole response-size classes, because the guest wall clock is microsecond-resolution andnanosis therefore always a multiple of 1000. It does not:as_nanos()since epoch is a ~61-bit monotonically advancing value, and% 1_000_000folds the high bits back in, so residues walk the whole bucket range across requests. Measured at n=200,000 per size against the binomial 3σ band, with the clock spread over a realistic arrival window: fixed sizes of 200 B, 4 KB, 64 KB, 100 KB, ~1 MB and 1.5 MB all land inside the band at bothaccess_sample_rate = 0.05and0.01; 0/400 random sizes are ever starved. Freezing the clock inside a single millisecond does produce an apparent bias, but that is an artifact of collapsing the reachable residue set, not a property of the sampler. No change wanted here —sampled_in's "approximately even, not provably unbiased" doc comment is accurate as written. -
📌
std::time::Instant::now()still traps onwasm32-unknown-unknownin ~15 pre-existing core sites. This PR correctly moved its own timing paths toweb_time, and #1075 tracks two of the neighbours — but the set is wider than that issue records:crates/trusted-server-core/src/auction/orchestrator.rs(lines 101, 295, 332, 375, 509, 581, 978, 1045, 1348),crates/trusted-server-core/src/auction/telemetry.rs:178,202,crates/trusted-server-core/src/integrations/datadome/protection_scope.rs:405,435, andcrates/trusted-server-core/src/publisher.rs:2259,4734(the template-cache TTL sites the new comments explicitly point at). Out of scope to change here; worth widening #1075's scope so the Cloudflare auction, DataDome-scope and template-cache-TTL paths are not left believed-covered. -
📝 The Axum adapter now hand-rolls
AxumDevServer's serve loop.crates/trusted-server-adapter-axum/src/main.rsreplacesAxumDevServer::with_config(router, config).run()with a localrun/servepair. Diffed against the pinned rev (edgezerotagv0.0.4,9e661ae,crates/edgezero-adapter-axum/src/dev_server.rs:274-321) and it is faithful:AxumDevServerConfighas exactlyaddrandenable_ctrl_cand both are honoured,serve_with_storeswithStores::default()inserts nothing extra, and theRouter::new().fallback_service(service_fn(...))+into_make_service_with_connect_info::<SocketAddr>()shape matches line for line. Only deviation istokio::net::TcpListener::bindwhere upstream binds astdlistener and callsfrom_std— behaviourally equivalent here. No finding against the code as written; the note is drift risk, since this copy will silently diverge at the nextedgezerorepin. Upstreaming a service-layer hook (AxumDevServer::with_service_layer) would let the fork go away. -
📝 Phase spans are not disjoint, so header subtimings do not partition
ts-total.Phase::Streamdeliberately encloses the in-stream auction wait —stream_drive_records_stream_ms_covering_the_in_stream_auction_waitassertsstream_ms >= auction_wait_ms— and every span is RAII across.await, so it measures wall clock including suspension rather than work. Both are the right choices for stall diagnosis, butdocs/guide/configuration.md's new "Observability" section reads as if the entries were a breakdown. One sentence saying the entries may overlap and do not sum tots-totalwould stop an operator drawing the wrong conclusion from the header. -
📝
ObservabilitySettings' rollback doc attributes the protection to the wrong attribute.crates/trusted-server-core/src/settings.rssays ofObservabilitySettings: "this struct denies unknown fields, so an older binary loading a config blob carrying an[observability]table it does not know would reject it, breaking rollback." The mechanism that actually makes rollback unsafe is#[serde(deny_unknown_fields)]onSettingsitself (settings.rs:2653) in the older binary;ObservabilitySettings' own attribute only rejects unknown keys inside the table in the new binary. The conclusion and theskip_serializing_if = "ObservabilitySettings::is_default"guard are both correct — only the stated reason is off, and it would mislead the next reader who tries to relax either attribute.
Verification performed
Scratch worktree at PR head 38043d74:
| Gate | Result |
|---|---|
gh pr checks 1074 |
19/19 PASS, 0 failing, 0 pending |
cargo fmt --all -- --check |
PASS |
cargo clippy-fastly / -axum / -cloudflare / -cloudflare-wasm / -spin-native / -spin-wasm |
PASS (all six) |
cargo test-fastly |
PASS — 2450 passed, 0 failed, 10 ignored |
cargo test-axum |
PASS — 40 passed, 0 failed |
cargo test-cloudflare |
PASS — 39 passed, 0 failed |
cargo test-spin |
PASS — 79 passed, 0 failed |
| parity suite | PASS — 13 passed, 0 failed |
npx vitest run |
PASS — 45 files, 871 tests, 0 failed |
Could not verify
- Fastly Compute production wall-clock granularity — the microsecond resolution behind the sampler check was measured under Viceroy on
wasm32-wasip1, not on Fastly Compute. The clean result holds for any granularity finer than a millisecond, so this does not change the conclusion. - Cloudflare
web_time::Instantbehaviour on workerd — no workerd here; taking the description's word that it was verified. Worth noting Workers deliberately freezes clocks between I/O, so CPU-only phases will likely read0there. Harmless while Cloudflare does not emit, but the collected values are not usable as-is if emission is turned on later. - Tinybird schema deployment state —
ENGINE_SORTING_KEYchanged incompatibly from the reserved schema; whetheraccess_logs_rawwas ever deployed remotely (and so whether this needs a versioned replacement plus cutover rather than an in-place edit) is a live-account question. Already disclosed as a rollout precondition; notbCLI here. - Fronting delivery layer
Server-Timingpass-through — needs a staging deploy; already disclosed. emit_access_eventagainst real Tinybird — exercised only through the recording double, so the live ingest contract is unverified.
The first path segment is only a section name when the path has depth: under a /%postname%/ permalink structure every article is a single-segment path, so keeping those segments verbatim put full article slugs into the 30-day dataset, against spec section 9. Depth is now required for a named template; single-segment paths, root landing pages included, bucket to /other/*. Route slicing keeps route_class and multi-segment templates like /news/*.
The bucket-quantized sampler truncated rates below one in a million to a zero threshold (silently emitting nothing) and quantized other low rates downward while rows still carried the configured rate, biasing the sum(1.0 / sample_rate) volume estimator. Its no-rand premise was also wrong: rand::thread_rng() is WASI-backed on this target and the EC generation path already relies on it. The sampler is now a direct uniform-roll comparison, and the roll gates on the rate stored in the snapshot itself, so emission probability and the row's sample_rate column cannot diverge; the divergence guard and its tests are removed. Also per review: the settings-reload fallback in the post-send path could never emit (no snapshot exists when settings were absent) and is removed; the dead_code allow on DeliveryOutcome narrows to the one collected-but-unemitted field; and the post-send ordering test is narrowed to the leg it actually proves, that request_elapsed is stamped when send returns.
On origin failure with a dispatched auction, the origin span guard stayed alive through the emit_abandoned_auction await, so ts-origin and origin_ms absorbed Tinybird emission time. The span now closes when the send resolves, before either branch, with an error-path regression test.
- Pin HEADER_PHASES against Phase::header_name() in the phase-index test, closing the second hand-synced list. - Add RouteClass::Asset to the snake_case rendering test; rename the lowercasing test to say what it does. - Give the Axum adapter a named, fully configured construction path (TrustedServerApp::dev_server_service) so server_timing_enabled is never silently discarded; the tuple API is private now. - Document that Server-Timing is client-visible when enabled, in the configuration guide's observability section. - Replace stale event_date references in the spec, plan, and dashboard guidance with the toDate(event_ts) sorting-key expression, and state the single-segment rejection rule in spec section 9.
…ming # Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-adapter-fastly/src/main.rs # crates/trusted-server-adapter-fastly/src/middleware.rs # crates/trusted-server-core/src/publisher.rs # crates/trusted-server-core/src/settings.rs # docs/guide/configuration.md # trusted-server.example.toml
The access emitter carried the configured body limit without enforcing it, allowing oversized rows to bypass the intended transport safeguard.
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
Approved at c6235b91fcd247a57d27fa8703734340d85d0c22. The timing, terminal header, and Tinybird transport controls passed focused validation. One medium-severity route-normalization finding is noted inline.
| } | ||
|
|
||
| if has_more_depth { | ||
| format!("/{lowered}/*") |
There was a problem hiding this comment.
Deep publisher paths can still retain usernames, slugs, and tokens: Any path with depth greater than one preserves its first segment after only character, length, and digit checks. Depth does not prove that the segment is a section name. Publisher fallback applies this function to every unmatched path at crates/trusted-server-adapter-fastly/src/app.rs:927.
With access telemetry enabled, paths such as /alice/profile or /hiv-diagnosis/comments retain caller-controlled content in route_template for the datasource's 30-day TTL. Attackers can also generate high-cardinality 32-character alphabetic segments. Direct execution of the production function produced /alice/profile -> /alice/*, /hiv-diagnosis/comments -> /hiv-diagnosis/*, and /aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/x -> /aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/*.
Suggested fix: Preserve a first segment only when it appears in an operator-configured allowlist of known section roots. Otherwise emit /other/*. If no configuration is desired, collapse all publisher fallback paths to one fixed template. Add deep short-segment adversarial tests.
prk-Jr
left a comment
prk-Jr
left a comment
There was a problem hiding this comment.
Summary
The timing model is thoughtfully structured: phase accumulation is saturating, response extensions keep adapter/core boundaries typed, and origin timing closes before abandonment telemetry. The telemetry schema, producer, fixture, and forwarding query are also broadly aligned.
I found five blocking issues and four non-blocking follow-ups. Two one-click suggestions were scratch-tested; only the Fastly streaming and fixture suggestions remained byte-for-byte unchanged after their checks. The rollback replacement was rewritten by the docs formatter, so I am leaving that finding as prose instead of posting an unverified patch.
Blocking findings
- 🔧 Publisher route telemetry retains attacker-controlled path content. For publisher fallbacks at depth two or greater,
publisher_route_templatepreserves any short allowlisted first segment. Requests such as/alice/ordersor sensitive-looking arbitrary prefixes can therefore become 30-day Tinybird dimension values. Please collapse non-root publisher fallbacks to a fixed template such as/other/*, or gate preserved segments behind an explicit operator-controlled allowlist. This is the same issue already tracked in the unresolved current-head thread: #1074 (comment) - 🔧 The documented rollback sequence can fail startup or unexpectedly enable auction telemetry. The
[tinybird]access fields are not all new and the nested Tinybird schema does not reject every unknown field. Older binaries rejectaccess_enabled = true, ignoreauction_enabled, and interpretenabled = trueas auction telemetry. Please document a compatibility config pushed before rollback: remove[observability], setaccess_enabled = false, and setenabled = falsefor access-only deployments before rolling back the binary. - 🔧 The
max_body_bytescontract is inconsistent. The guide, settings docs, design spec, and implementation plan describe a positive value, while runtime validation rejects values below 1024. Please make every published contract say>= 1024(or change validation and tests if the intended contract is actually> 0). - 🔧 The required CodeQL check is failing. The current head reports 11 high-severity
rust/cleartext-loggingalerts. The inspected paths suggest the analyzer may be following the Tinybird secret-key-name validation rather than an actual token value, but the merge gate remains red. Please fix/suppress the flow or dismiss the alerts with a documented rationale before merge.
The remaining blocking issue, buffering the unused Tinybird response body, has an inline verified suggestion.
Non-blocking findings
- 🤔 The Axum timing test only proves that the outer wrapper emits
ts-total; it does not prove that request extensions survive the adapter boundary and phase timings recorded in the core handler return to the wrapper. - 🏕 The complete Fastly post-send order—elapsed snapshot, pull sync, then access telemetry—does not have the instrumented sequence regression test called for by the implementation plan.
- 📝 Snapshot lifecycle documentation is stale. The implementation now constructs the snapshot only when access telemetry is enabled, while the core comment, design spec, implementation plan, and PR description still say it is unconditional.
- ⛏ The Tinybird fixture uses a reserved-for-testing-style publisher hostname rather than this repository's example-domain convention; an inline suggestion updates it.
Checks
- GitHub: 18 checks pass;
CodeQLfails on headc6235b91fcd247a57d27fa8703734340d85d0c22. - Scratch verification for the Fastly streaming suggestion:
cargo fmt --all -- --check; all Fastly, Axum, Cloudflare, Cloudflare WASM, Spin native, and Spin WASM clippy aliases;cargo test-fastly,cargo test-axum,cargo test-cloudflare,cargo test-spin, and cross-adapter parity all passed. - Scratch verification for the rollback text: docs formatting and the same Rust test/clippy matrix passed, but docs formatting changed the proposed bytes, so no GitHub suggestion is attached.
- Scratch verification for the fixture suggestion: JSON parsing, the expected domain assertion, and
git diff --checkpassed without changing the proposed bytes.
| ); | ||
|
|
||
| let response = client | ||
| .send(PlatformHttpRequest::new(request, backend_name)) |
There was a problem hiding this comment.
🔧 Blocking: this request uses the default buffered response mode even though the Tinybird response body is never consumed. On Fastly, buffered conversion calls take_body_bytes() before enforcing the 10 MiB limit, so a chunked/no-length response can be fully materialized; ordinary successful responses also waste memory. Please stream the response, and add a recording-client regression test that asserts stream_response is set.
This exact change passed the full scratch test and clippy matrix without byte drift.
| .send(PlatformHttpRequest::new(request, backend_name)) | |
| .send(PlatformHttpRequest::new(request, backend_name).with_stream_response()) |
|
|
||
| ### Deploy and rollback ordering | ||
|
|
||
| ::: warning `Settings` rejects unknown fields; order matters |
There was a problem hiding this comment.
🔧 Blocking: this rollback guidance misstates the compatibility boundary. The Tinybird access fields already existed and the nested schema accepts unknown fields, while older binaries reject access_enabled = true, ignore auction_enabled, and treat enabled = true as enabling auction telemetry. Please document pushing a compatibility config first: remove [observability], set access_enabled = false, and set enabled = false for an access-only deployment before rolling back the binary.
I scratch-tested the approved replacement, but the docs formatter rewrote its line wrapping, so I am intentionally leaving this as prose rather than attaching a one-click suggestion that fails the repository's exact format gate.
| } | ||
|
|
||
| #[tokio::test(flavor = "multi_thread", worker_threads = 2)] | ||
| async fn axum_emits_header_on_private_response() { |
There was a problem hiding this comment.
🤔 This test verifies the outer timing wrapper (ts-total) but never exercises the extension round-trip that carries core phase timings back through EdgeZeroAxumService. Could the handler record one named phase (for example filter) and assert the response contains the corresponding ts-* metric? That would catch a future adapter conversion that accidentally drops request extensions.
| &partner_registry, | ||
| &ec_state, | ||
| ); | ||
| emit_access_telemetry_after_send(&settings, &outcome, &timings); |
There was a problem hiding this comment.
🏕 The implementation plan calls for an instrumented test of the complete post-send sequence, but existing coverage only locks down the elapsed-before-pull-sync leg. Please add a sequence assertion covering elapsed snapshot → pull sync → access telemetry, so telemetry cannot later move ahead of EC synchronization unnoticed.
| @@ -0,0 +1 @@ | |||
| {"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} | |||
There was a problem hiding this comment.
⛏ Use the repository's example-domain convention in the committed fixture. This exact fixture edit passed JSON validation and git diff --check without byte drift.
| {"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} | |
| {"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"publisher.example.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} |
aram356
left a comment
aram356
left a comment
There was a problem hiding this comment.
Summary
Adds always-on per-request phase timing with a Server-Timing header and a sampled access-telemetry sink to Tinybird. The engineering is careful — a single shared emission gate across adapters, sampling that cannot diverge from the row's own sample_rate, a body limit that rejects rather than truncates, exact schema/producer alignment, and 123 new tests.
Three blocking items: a PII leak in the publisher route classifier, a config doc that states a constraint the code rejects, and a failing CodeQL gate.
2 of the inline comments below carry a one-click GitHub
suggestion— use Commit suggestion (or Add suggestion to batch) to apply them. The remaining comments describe the fix in prose because the change spans multiple files, needs new config plumbing, or is a question.
I verified the cache-safety gate separately and it is sound: an isolated probe showed Cache-Control: private plus Surrogate-Control: max-age=600 passing append_server_timing_if_private, but on the real path enforce_uncacheable_cache_privacy strips every edge-cache header immediately before the freeze point, and enforce_uncacheable_cache_privacy_handles_late_filter_headers already pins that exact combination. Not a finding.
Blocking
🔧 wrench
publisher_route_templateemits user data verbatim at depth >= 2 — see inline atcrates/trusted-server-core/src/access_telemetry.rs:194- Docs state a
max_body_bytesfloor the code rejects — see inline atdocs/guide/configuration.md:2010 - CodeQL is failing with 11 new high-severity alerts — see Cross-cutting below
Non-blocking
♻️ refactor / 🤔 thinking / 📝 note / ⛏ nitpick
TimedKvStoredoes not forwardexists— see inline atcrates/trusted-server-core/src/platform/timed_kv.rs:45- 401 write-back clobbers a resolved geo state — see inline at
crates/trusted-server-adapter-fastly/src/middleware.rs:109 - Poisoned mutex is indistinguishable from
WouldBlock— see inline atcrates/trusted-server-core/src/request_timing.rs:139 - Integration-proxy branch degrades an already-bounded route — see inline at
crates/trusted-server-adapter-fastly/src/app.rs:876 auction_enabledlacksskip_serializing_if, so rollback silently re-enables auction telemetry — see inline atcrates/trusted-server-core/src/settings.rs:1794- Access sink omits the trailing newline the auction sink appends — see inline at
crates/trusted-server-adapter-fastly/src/tinybird.rs:348 - Axum reads
server_timing_enabledonce at startup — see inline atcrates/trusted-server-adapter-axum/src/app.rs:625 - The outer-wrapper rationale does not hold in this application — see inline at
crates/trusted-server-adapter-axum/src/timing.rs:14 ts-totalstarts afterinit_logger()and the JA4 short-circuit — see inline atcrates/trusted-server-adapter-fastly/src/main.rs:126- Fixture uses a real registrable domain — see inline at
tinybird/fixtures/access_logs_raw.ndjson:1
Cross-cutting / body-level findings
-
🔧 CodeQL is failing with 11 new high-severity alerts.
CodeQLreports "11 new alerts including 11 high severity security vulnerabilities", allCleartext logging of sensitive information. Branch protection does not require this check, but CLAUDE.md treats it as a PR gate. Reading the alerts, they look like false positives: they namevalidate_tinybird_secret(...)as a taint source, but that function takes a Secret Store key name (not a secret value), never returns it, and only formats the setting name into an error message. The flagged sinks are almost all in files this PR does not touch (adapter-axum/src/platform.rs,adapter-fastly/src/management_api.rs,auction/orchestrator.rs,response_privacy.rs,storage/kv_store.rs) — the new validator appears to have introduced a source that lights up pre-existing log sites. Please either dismiss the alerts with that rationale recorded, or rename the parameter to make the key-name-vs-secret-value distinction legible to the analysis. Leaving a red gate red without a recorded rationale is the part worth avoiding. -
📝 Test-coverage gaps worth closing alongside the findings above. (1) No Axum-layer test that
server_timing_enabled = falsesuppresses the header — all three tests intiming.rsconstructTimingService::new(..., true), so a refactor that drops the flag read would ship silently. (2) No test that a response with noCache-Controlheader at all is excluded; that is the fail-closed case and the single most security-relevant assertion in the feature. (3) No classifier test for a short user-data first segment at depth >= 2 (the case in the first finding) — every existing rejection case is depth-1 or caught by the 32-char cap. -
📝
DeliveryResult::Partial/Erroris collected but never emitted, behind#[allow(dead_code)]atmain.rs:533-538. The PR description discloses this as intentional groundwork, which is fine — worth confirming a follow-up issue exists so it does not settle in as permanently dead code.
CI Status
- integration tests (Fastly EC lifecycle): PASS
- browser integration tests: PASS
- integration tests: PASS
- CodeQL: FAIL (not required by branch protection)
- Analyze (actions): PASS
- Analyze (rust): PASS
- Analyze (javascript-typescript): PASS
- prepare integration artifacts: PASS
- format-typescript: PASS (required)
- format-docs: PASS (required)
- cargo test: PASS (required)
- cargo fmt: PASS (required)
- cargo test (axum native): PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- cargo test (cross-adapter parity): PASS
- cargo test (ts CLI, native): PASS
- vitest: PASS
| } | ||
|
|
||
| if has_more_depth { | ||
| format!("/{lowered}/*") |
There was a problem hiding this comment.
🔧 wrench — This branch emits the first path segment verbatim, so publisher paths of depth >= 2 carry user data into a 30-day dataset.
The single-segment rejection just below is the real privacy control, and it only applies at depth 1. At depth >= 2 the segment passes on a character allowlist, a 32-char cap, and a 7-digit cap alone. I extracted this function and ran it:
/jane-doe/posts -> /jane-doe/*
/johnsmith1985/settings -> /johnsmith1985/*
/how-to-treat-my-hiv-today/comments -> /how-to-treat-my-hiv-today/*
/abcdefabcdefabcdefabcdefabcdefab/x -> /abcdefabcdefabcdefabcdefabcdefab/*
/8f3a9c2b/x -> /8f3a9c2b/*
Concretely: /{username}/posts and /{username}/settings are the standard Substack / Medium / Ghost / forum shape, so every visited username lands in Tinybird. A sub-32-char health slug does too — the test at line 412 asserts /how-to-treat-my-recent-hiv-diagnosis is rejected by length (36 chars), so the same slug at 25 chars sails through, and that test reads as stronger than it is. A 32-char all-letter hex id has zero digits and passes both bounds; an 8-hex-char session id has 4 digits and passes too.
This contradicts the module doc's "bounded, content-free route identity" and RouteMetadata.route_template's "Bounded, content-free", and it undercuts the schema change that deliberately replaced the raw path column with route_template. Because the column is a plain String rather than LowCardinality, it is also an unbounded-cardinality vector.
Proposed fix (apply manually — needs a new config field plus plumbing, so it cannot be a single-file suggestion): depth is not a sufficient signal for "this segment is a section name". Gate the depth >= 2 branch on an operator-configured allowlist of section names, defaulting to empty so every publisher path collapses to /other/* until a publisher opts in:
if has_more_depth && section_allowlist.contains(&lowered) {
format!("/{lowered}/*")
} else {
"/other/*".to_owned()
}That is the construction that makes the content-free claim true, and it bounds cardinality enough that the column could become LowCardinality. Whichever fix you choose, please add the five inputs above as test cases — right now no test exercises a short user-data first segment at depth >= 2.
| `secret_store`, `access_dataset`, `access_token_secret`, and a positive | ||
| `max_body_bytes`) to already be set. |
There was a problem hiding this comment.
🔧 wrench — This says "a positive max_body_bytes", but TinybirdSettings::prepare_runtime rejects anything below 1024:
if self.max_body_bytes < 1024 {
return Err(Report::new(TrustedServerError::Configuration {
message: "tinybird.max_body_bytes must be at least 1024".to_owned(),
}));
}An operator who reads this and sets max_body_bytes = 512 to be conservative on an edge budget gets a config that fails to load — a deploy-time outage caused by the doc promising the value was legal.
| `secret_store`, `access_dataset`, `access_token_secret`, and a positive | |
| `max_body_bytes`) to already be set. | |
| `secret_store`, `access_dataset`, `access_token_secret`, and a | |
| `max_body_bytes` of at least 1024) to already be set. |
(Verified: applied in a scratch worktree, npm run format in docs/ passes.)
| | Field | Type | Required | Default | Description | | ||
| | -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | | ||
| | `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | | ||
| | `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | | ||
| | `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | | ||
| | `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | | ||
|
|
||
| **Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird | ||
| sinks separately so that turning on one does not silently turn on (or leave |
There was a problem hiding this comment.
🔧 wrench — The section says "the keys below cover the access-telemetry sink", but three keys it requires are missing from the table: access_dataset, access_token_secret, and max_body_bytes all appear in the prose and the TOML example with no type or default given.
This suggestion adds the three rows. Column padding is Prettier's canonical alignment for the widened table — the whole table has to be replaced rather than just the new rows, or format-docs (a required check) fails.
| | Field | Type | Required | Default | Description | | |
| | -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | | |
| | `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | | |
| | `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | | |
| | `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | | |
| | `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | | |
| **Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird | |
| sinks separately so that turning on one does not silently turn on (or leave | |
| | Field | Type | Required | Default | Description | | |
| | --------------------- | ------- | ------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------- | | |
| | `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | | |
| | `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | | |
| | `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | | |
| | `access_dataset` | String | Yes, when `access_enabled` | `access_logs_raw` | Access-log Events API datasource name | | |
| | `access_token_secret` | String | Yes, when `access_enabled` | `tinybird_access_append_token` | Secret Store key holding the access APPEND token | | |
| | `max_body_bytes` | Integer | No | `1048576` | Maximum NDJSON request body size; must be at least 1024 | | |
| | `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | |
(Verified: applied in a scratch worktree, npm run format in docs/ passes.)
| } | ||
|
|
||
| #[async_trait(?Send)] | ||
| impl PlatformKvStore for TimedKvStore<Arc<dyn PlatformKvStore>> { |
There was a problem hiding this comment.
♻️ refactor — This impl covers five of the trait's six methods; exists is not forwarded, so the decorator inherits the trait's default body:
async fn exists(&self, key: &str) -> Result<bool, KvError> {
Ok(self.get_bytes(key).await?.is_some())
}Three adapters override exists with that same get-and-discard shape, but the Spin adapter overrides it with a native metadata-only call (edgezero-adapter-spin/src/key_value_store.rs:90). So wrapping a Spin store in TimedKvStore silently downgrades a cheap existence probe into a full value transfer and heap allocation — for a large consent blob, a real latency and memory regression caused purely by decorating. The trait doc warns about exactly this: "Backends that support a cheaper existence check should override this."
I checked and there is currently no KV exists caller in the repo, so this is latent rather than live — but transparency is the decorator's entire contract, and this will bite whoever adds the first caller.
Proposed fix (apply manually — adding a method to the impl block sits outside this hunk):
async fn exists(&self, key: &str) -> Result<bool, KvError> {
let _span = self.timings.span(Phase::EcKv);
self.inner.exists(key).await
}Worth a delegation test too: a stub whose exists returns true while get_bytes returns None fails today and passes after the fix.
| let resolved_state = match &geo_info { | ||
| Some(geo) => GeoLookupState::Resolved(geo.clone()), | ||
| None => GeoLookupState::Attempted, | ||
| }; | ||
| response.extensions_mut().insert(resolved_state); |
There was a problem hiding this comment.
🤔 thinking — This write-back is unconditional, but resolve_geo_for_response returns None for a 401 before it consults carried. So on a 401 a carried GeoLookupState::Resolved(geo) is overwritten with Attempted, and build_access_telemetry_snapshot then emits country: "unknown" — pinned by access_snapshot_treats_attempted_geo_lookup_as_unknown_country.
The comment directly above says the write-back exists so telemetry sees what was actually looked up "rather than the stale carried-in state". On the 401 path the carried state is what was actually looked up, and the write-back is what makes it stale.
No leak and no extra hostcall (Attempted is terminal), so this is data loss rather than a defect — and suppressing country on 401 rows may well be what you intended. But the code currently reads as if the opposite were happening.
Proposed fix (apply manually — best factored into the shared helper so main.rs:404-411 gets it too, which puts it outside this hunk):
if response.status() != StatusCode::UNAUTHORIZED {
let resolved_state = match &geo_info {
Some(geo) => GeoLookupState::Resolved(geo.clone()),
None => GeoLookupState::Attempted,
};
response.extensions_mut().insert(resolved_state);
}If suppression is intended, a comment saying so plus a test asserting it would settle it.
| .change_context(TrustedServerError::Proxy { | ||
| message: "Tinybird backend registration failed".to_owned(), | ||
| })?; | ||
| let request = build_access_events_request(target, row, auth_header)?; |
There was a problem hiding this comment.
🤔 thinking — The access row goes out as a bare JSON object under Content-Type: application/x-ndjson, with no trailing newline. The auction path's to_ndjson appends \n after every row and its test asserts ends_with('\n') with the message "should send newline-delimited JSON". The access test at line 952 instead asserts the body is the row verbatim, locking the divergence in.
Tinybird's Events API tolerates a final row without a terminator, so this very likely works today. Flagging it because two sinks posting different framing to the same API under the same content type is the kind of inconsistency that only surfaces when the endpoint tightens, and the verbatim assertion would mask it.
Proposed fix (apply manually — touches emit_access_event and its test together): append \n before the length check, so the limit still accounts for the byte, and update the assertion to match the auction sink's.
| } | ||
| }; | ||
|
|
||
| let server_timing_enabled = state.settings.observability.server_timing_enabled; |
There was a problem hiding this comment.
📝 note — This reads server_timing_enabled once at startup and bakes it into the service, while the Fastly adapter reads it per request from a freshly built settings snapshot. The code comment just above documents the divergence, but docs/guide/configuration.md does not — and that section explicitly discusses Axum adapter behavior.
Minor for a dev server, but an operator flipping the flag and seeing nothing change has no way to know a restart is required. One sentence in the [observability] section would close it.
| //! middleware never sees those responses. By the time a response reaches | ||
| //! this layer -- after `RouterService::oneshot` inside | ||
| //! `EdgeZeroAxumService::call` has already converted any dispatch error into | ||
| //! a plain response -- every response is covered uniformly, router-generated | ||
| //! or not. | ||
| //! | ||
| //! `/health` is excluded by path match before a |
There was a problem hiding this comment.
📝 note — The module doc justifies the outer-wrapper position on router-generated 404/405 responses bypassing RouterBuilder::middleware. That case is essentially unreachable in this application: build_router registers a catch-all across every publisher method, so every path and all seven methods hit the fallback handler and run through the middleware chain.
for method in publisher_fallback_methods() {
router = router.route("/", method.clone(), fallback.clone());
router = router.route("/{*rest}", method, fallback.clone());
}A router-generated 404 needs a path miss (impossible against /{*rest}), and a 405 needs a method outside those seven (TRACE, CONNECT). The test at line 170 does exercise the mechanism, but against an empty RouterService::builder().build() rather than this application's router.
The outer position is still the right call — it is the honest terminal freeze point and it survives future routing changes. Only the stated rationale overshoots. Worth rewording so a reader does not go looking for the 404 path that motivated it.
| return; | ||
| } | ||
| }; | ||
| let timings = RequestTimings::new(); |
There was a problem hiding this comment.
📝 note — t0 is set here, after FastlyRequest::from_client(), the health check, logging::init_logger(), and the JA4 short-circuit have already run. init_logger typically registers a Fastly log endpoint, which is a hostcall, so ts-total systematically undercounts real request time by that prologue.
Not wrong, but an operator reading ts-total will reasonably take it as wall-clock-from-accept. Either move RequestTimings::new() ahead of init_logger() and pass it into edgezero_main, or note the boundary in the time_elapsed_ms doc comment so the exclusion is discoverable.
| @@ -0,0 +1 @@ | |||
| {"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} | |||
There was a problem hiding this comment.
⛏ nitpick — publisher_domain is test-publisher.com, a real registrable domain. CLAUDE.md requires fictional data in fixtures ("for urls use: example.com domains only"), and the sibling fixture committed in this same directory, tinybird/fixtures/auction_events_raw.ndjson, already uses test-publisher.example.
Not offered as a suggestion because the fix is a single token inside a 26-field JSON line and a one-click replacement of the whole row would be more error-prone than editing it in place: change "publisher_domain":"test-publisher.com" to "publisher_domain":"test-publisher.example".
4 participants
Closes #1068. Implements the design in #1069 (
docs/superpowers/specs/2026-08-24-request-phase-timing-design.md); the implementation plan and the spec ride in the branch.What this adds
RequestTimings(core): always-on per-request phase collection;try_lock-only, saturating, infallible by construction.Server-Timingheader (ts-total,ts-appbuild,ts-filter,ts-geo,ts-kv,ts-origin,ts-template-cache) emitted at the send freeze point immediately beforeinto_parts(), gated byobservability.server_timing_enabledand restricted to conclusively private (private/no-store) responses so no shared cache can replay timings.TimedKvStoredecorator implementing bothPlatformKvStoreandEcKvStore(latency-only; reads no payloads). Pull-sync stores are explicitly untimed.GeoLookupStateresponse extension (NotAttempted/Attempted/Resolved), 401 rule preserved.DeliveryResult::{Complete, Partial, Error}), auction-wait with explicit placement (in_streamat the seam,pre_headeron buffered paths), response bytes, and a post-bodyrequest_elapsed_msthat excludes pull-sync and telemetry.AccessTelemetrySnapshotbuilt unconditionally at the freeze point, coarse PII-safe route templates (allowlist-reject; adversarially tested with EC ids, emails, search terms), and a confirmed-delivery Tinybird sink (bounded await, 2xx-validated, sampled bytinybird.access_sample_rate) that runs after client delivery and after pull-sync.access_logs_rawschema aligned column-for-column with the row producer; sorting key(toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status); there is noevent_datecolumn (Tinybird's Events API requires a JSONPath on every column, which a derived-default column cannot carry).docs/guide/configuration.md.Review process
Eleven plan tasks, each implemented and passed an independent task-scoped review; two task-level fix rounds (settings validation coverage; a schema/producer nullability mismatch caught before it could quarantine rows at ingestion); a final whole-branch review on the full 13-commit diff followed by one fix wave (method-token normalization, a zero-sample-rate guard, geo write-back symmetry) and a clean scoped re-review.
Known limitations and rollout preconditions (disclosures)
access_logs_rawwas ever deployed remotely: the sorting key changed incompatibly from the reserved schema, so a deployed datasource means a versioned replacement with cutover, not an in-place edit. Panel queries needEXPLAINvalidation against the new key.[observability]table.ts-appbuild); Cloudflare and Spin collect but do not emit in v1. An earlier revision of this branch usedstd::time::Instant, which panics onwasm32-unknown-unknownand trapped every Cloudflare publisher request (the failingintegration testsruns on this branch); the timing paths now useweb_time::Instant, verified against the real workerd runtime locally.DeliveryResultis collected but not yet emitted on any surface (intentional groundwork).Partialsemantics are error-based only, per an explicit owner ruling: clean-but-early source truncation is out of scope permanently.request_elapsed_msis stamped beforesend_to_client(no drive to time); streaming responses, the case that matters for stall diagnosis, include the full drive.access_sample_rate = 1.0is a diagnosis setting, not a steady state.Generated with Claude Code